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We present new direct constraints on a general Wtb interaction using data corresponding to an 
integrated luminosity of 5.4 fb _1 collected by the DO detector at the Tevatron pp collider. The 
standard model provides a purely left-handed vector coupling at the Wtb vertex, while the most 
general, lowest dimension Lagrangian allows right-handed vector and left- or right-handed tensor 
couplings as well. We obtain precise limits on these anomalous couplings by comparing the data to 
the expectations from different assumptions on the Wtb coupling. 
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The top quark was discovered in 1995 at the Teva- 
tron 0, 0] via the pair production mode involving strong 
interactions. In 2009, the electroweak production of the 
top quark was observed by the DO and CDF collabora- 
tions 0, IU • At the Tevatron, the dominant production 
modes for single top quark are the s-channel ("£&") [H[ 
and t-channel ( "tqb" ) 0, 0] processes illustrated in Fig- 
ure [1] A third process, usually called "associated pro- 
duction" in which the top quark is produced together 
with a W boson, has a small cross section at the Teva- 



(a) (b) 




FIG. 1: Tree level Feynman diagrams for (a) tb and (b) tqb 
single top quark production. 
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tron compared to the tb and tqb processes [8|. Recently, 
we presented improved measurements of the single top 
quark production cross sections Q and the observation 
of i-channcl single top quark production [l(| ■ 

The large mass of the top quark implies that it has 
large couplings to the electroweak symmetry breaking 
sector of the standard model (SM) and may have non- 
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standard interactions with the weak gauge bosons. Single 
top quark production provides a unique probe to study 
the interactions of the top quark with the W boson. 

The most general, lowest dimension, CP-conserving 
Wtb vertex is given by 
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C = -JLbrfiLvPL + RvPrW; 



V2 Mi 



(L T P L + R T P R )tw: 



(1) 



where Myy is the mass of the W boson, q v is the W boson 
four-momentum, P L = (1 — ~f§)/2 is the left-handed pro- 
jection operator, Pr = (l+7s)/2 is the right-handed pro- 
jection operator, L v _t = Vtb-fhv, T an d Rv,t = V t b-fR V}T - 
The form factor Jl v (/l t ) represents the left-handed 
vector (tensor) coupling, fn v (Jr t ) represents the right- 
handed vector (tensor) coupling, and Vtb is the Cabibbo- 
Kobayashi-Maskawa matrix element. In the SM, the 
Wtb coupling is left-handed with Ly = \Vtb\ — 1 and 
Ry = Lt = Rt = 0. The magnitudes of the right- 
handed vector coupling and the tensor couplings can be 
indirectly constrained by the measured branching ratio of 
the b — > S7 process [l2[ . Measurements of top quark de- 
cays in it production, e.g. the W boson helicity [l3, 14|, 
can directly constrain the Lorentz structure of the Wtb 
vertex [l5|. Assuming single top quarks are produced 
only via W boson exchange, the single top quark cross 
section is directly proportional to the square of the effec- 
tive Wtb coupling. Moreover, the event kinematics and 
angular distributions are also sensitive to the existence 
of anomalous top quark couplings 1|| 17 1. Therefore, di- 
rect constraints on anomalous couplings can be obtained 
by measuring single top quark production (l8j | . 

This analysis uses the same data, event selection, and 
background modeling as the recent single top quark cross 
section measurements [tj [l(J. We perform a study of 
anomalous Wtb couplings and obtain substantial im- 
provements on the limits of these couplings following the 
general framework given in Ref . [l8| • Out of the four cou- 
plings (Ly, Lt, Ry, Rt), we consider three cases pairing 
the left-handed vector coupling with each of the other 
three couplings: (Ly, Ry), (Ly, Lt) and (Ly, Rt), and 
for each case we assume the other two non-SM couplings 
are negligible. This pairing allows us to limit the com- 
plexity of the analysis, while increasing the statistical 
power and sensitivity for the anomalous coupling under 
study. We assume that single top quarks are produced 
exclusively through W boson exchange. Therefore other 
single top quark production mechanisms, such as flavor- 
changing neutral current interactions ljj, 2(|, the decay 
of new scalar boson HHl , or the exchange of new vector 



boson 22, 23| are not considered here. We also assume 
that the Wtb vertex dominates top quark production and 
decay, i.e., \V t d\ 2 + \ V ts \ 2 < | V t b\ 2 - The results presented 
here supersede those contained in Ref. [l8j]. In addition 



to the analysis of additional data and to the improve- 
ments in the event selection and background modeling 
used for the results presented in Ref. [3, [l(| , we use an 
updated calculation of the cross section as a function 
of the anomalous couplings which addresses a mistake 
present in the original analysis [l8| (in that analysis the 
Rt and Lt couplings were accidentally swapped). 

We select single top quark events which are expected 
to contain exactly one isolated large transverse momen- 
tum (pt) electron or muon and large missing transverse 
energy (fir)- Events with 2, 3 or 4 jets are selected, 
and one or two of the jets are required to originate from 
the hadronization of long-lived b hadrons (b-.jets) as de- 
termined by a multivariate 6-tagging algorithm 24(. To 
increase the search sensitivity, we divide our data into 
six independent analysis channels, each with a different 
background composition and signal-to-background ratio. 
The channels are based on the number of identified b 
jets (1 or 2) and jet multiplicity (2, 3 or 4 jets). The 
signal selection efficiencies with different Wtb couplings, 
including branching fraction, trigger efficiencies and the 
6-tagging requirements, vary between 2.7% and 3.0% for 
tb and 1.9% and 2.2% for tqb production, estimated using 
Monte Carlo (MC) simulations. The "associated produc- 
tion" process gives a negligible contribution to this anal- 
ysis. 

Single top quark signal events with the SM and anoma- 
lous Wtb couplings are modeled using the COMPHEP- 
based MC event generator SINGLETOP [25( for a top quark 
mass m t = 172.5 GcV using the CTEQ6M [2(§ parton 
distribution functions. The anomalous Wtb couplings 
are taken into account in both production and decay in 
the generated samples. The event kinematics for both s- 
channel and ^-channel processes reproduce distributions 
from next-to-leading-order calculations [27, 28 1. The de- 
cay of the top quark and the resulting W boson are car- 
ried out in the SINGLETOP [25[ generator in order to pre- 
serve information about the spin of the particles. The 
theoretical cross sections for anomalous single top quark 
production (s+i-channel) with \V t b\ — 1 are 3.1 ± 0.3 pb 
if Irv = 1; 9 -4 ± 1-4 pb if /l t = 1 or f Rr = 1, and 
10.6 ± 0.8 pb if J Rt = f Lv = 1 0. All other cou- 
plings are set to zero when calculating these cross sec- 
tions. The SM single top quark production cross section 
is 3.3 ±0.1 pb [|. 

The main background contributions are those from 
W bosons produced in association with jets (W+jets), 
it production, and multijet production in which a jet 
with high electromagnetic content mimics an electron, or 
a muon contained within a jet originating from the decay 
of a heavy-flavor quark (b or c quark) appears to be iso- 
lated. Diboson (WW , WZ, ZZ) and -Z+jets processes 
add small additional contributions to the background. 
The it, W+]ets, and Z+jets events are simulated with 
the ALPGEN leading-log MC generator Q . The effect of 
anomalous Wtb couplings on kinematic distributions of 
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the tt background has been found to be negligible, thus 
only SM tt samples are considered in the analysis. Dibo- 
son processes are modeled using pythia §}()} . For all of 
the signal and background MC samples, PYTHIA is used 
to simulate parton showers and to model hadronization 
of all generated partons. The presence of additional pp 
interactions is modeled by events selected from random 
beam crossings matching the instantaneous luminosity 
profile in the data. All MC events are processed through 
a GEANT-based simulation [3l| of the DO detector and 
reconstructed using the same algorithm as data. Differ- 
ences between simulation and data in lepton and jet re- 
construction efficiencies and resolutions, jet energy scale, 
and 6-tagging efficiencies are corrected in the simulation 
by applying correction functions measured from separate 
data samples. The ti, Z+jets and diboson MC samples 
are scaled to their theoretical cross sections [32], [33| . We 
use data containing non-isolated leptons to model the 
multijet background. W+jets and multijet backgrounds 
are normalized by comparing the prediction for back- 
ground to data before 6-tagging. Details of the selection 
criteria and background modeling are given in Ref. 0] . 

The main contributions to the systematic uncertainty 
on the predicted number of events arise from the signal 
modeling, the jet energy scale (JES), jet energy resolution 
( JER) , corrections to 6-tagging efficiency and the correc- 
tion for jet-flavor composition in VT±jets events. These 
uncertainties affect the normalization of the distribu- 
tions, and in some cases (JES, JER, and 6-tagging) also 
change the differential distributions. There are smaller 
contributions due to limited statistics of the MC samples, 
uncertainties on the measured luminosity, and the trigger 
modeling. In addition, we also consider a signal cross sec- 
tion uncertainty (3.8% for tb and 5.3% for tqb) given by 
the NLO calculation. Details of systematic uncertainties 
are given in Ref. . Table U lists the numbers of events 
expected and observed for each process as a function of 
jet multiplicity. 

We use a multivariate analysis technique called 
Bayesian neural networks (BNN) [34j to separate the sig- 
nal from the backgrounds. The BNN discriminant is 
trained using the lepton and jets four-vectors, a two- 
vector for JpT , and variables that include lepton charge 
and 6-tagging information. In addition, four angular vari- 
ables are added based on top quark spin and W boson 
hclicity information to provide additional discriminating 
power. The total number of variables used in training 
for events with 2, 3, and 4 jets is 18, 22, and 26, respec- 
tively Three example distributions from some of the 
most sensitive variables are shown in Figure [2] the px 
spectrum of the lepton from the decay of the top quark 
and the cosine of the angles between the lepton, the lead- 
ing 6-tagged jet and the reconstructed top quark. 

For each of the three scenarios, we consider the anoma- 
lous coupling sample as the signal when training BNN 
discriminants: for (Ly, Ry), the signal is the single top 



TABLE I: Numbers of expected and observed events in 
5.4 fb _1 of integrated luminosity, with uncertainties includ- 
ing both statistical and systematic components. The single 
top quark contributions are normalized to their theoretical 
predictions. 



Source 






2 jets 


3 jets 


4 jets 


tb (f LT = 


1) 




756 ± 42 


344 ± 27 


103 ± 


15 


tqb (f LT = 


1) 




103 ± 5.8 


67 ± 6.3 


28 ± 


4.4 


tb (f R y = 


1) 




105 ± 6.0 


43 ± 3.8 


12 ± 


1.9 


tqb (fa v = 
tb (fn T = 


1) 




122 ± 7.2 


61 ± 5.3 


22 ± 


3.7 


1) 




730 ± 38 


316 ± 25 


92 ± 


14 


tqb (fn T = 


1) 




117 ± 6.2 


86 ± 8.6 


40 ± 


5.8 


tb (f Lv = 


/fir 


= 1) 


607 ± 31 


284 ± 21 


86 ± 


13 


tqb {fh v = 


fR T 


= 1) 


268 ± 15 


167 ± 16 


67 ± 


10 


tb (SM, h 


'V 


1) 


104 ± 16 


44 ± 7.8 


13 ± 


3.5 


tqb (SM, f Lv = 


1) 


140 ± 13 


72 ± 9.4 


26 ± 


6.4 


tt 






433 ± 87 


830 ± 133 


860 ± 


163 


W+jets 






3,560 ± 354 1,099 ± 169 


284 ± 


76 



Z+jets and dibosons 400 ± 55 142 ±41 35 ± 18 

Multijets 277 ± 34 130 ± 17 43 ± 5.2 

Total SM prediction 4,914 ± 558 2,317 ± 377 1,261 ± 272 
Data 4~881 2^307 L~283 



quark sample generated with fn v = 1; for (Ly, Rt), the 
signal is the sample generated with fn T — 1; for (Ly, 
Lt), the signal is the sample generated with /Y T = 1. 
The background includes the SM single top quark sample 
with Jl v = 1 and all the backgrounds described above. 
Each background component is represented in proportion 
to its expected fraction given by the background model. 
Figure [3] shows representative BNN discriminant output 
distributions for the three different scenarios with all six 
analysis channels combined. 

We follow a Bayesian statistical approach 0, [35], |3(| 
to compare data to the signal predictions given by dif- 
ferent anomalous couplings using BNN discriminant out- 
put distributions. We compute a two-dimensional (2D) 
posterior probability as a function of |Vj& • fh v \ 2 and 
\Vtb ■ fx\ 2 , where V tb • fx is one of the two non-SM cou- 
plings X = {Ry, Lt}- For these two cases the single top 
quark contribution is represented by a superposition of 
two samples: 

s=\V tb -f Lv \ 2 SL v + \V tb -fx\ 2 sx, (2) 

where s^ v (sx) are the mean expected count of single top 
quarks for the assumptions fi, v = 1 (fx = 1) and the 
other couplings are set to zero. In the (Ly, Rt) scenario, 
the two couplings interfere, and to account for the effect 
of the interference, the single top quark contribution is 
represented by the superposition of three samples: 

s = \V tb ■ f Lv \ 2 s Lv + \V tb ■ f RT \ 2 s RT + 

+ \Vtb ■ fL v \\V tb ■ fn T \(sL v R T — S L V — Sr t ), (3) 

where sr t is the mean count assuming a left-handed ten- 
sor coupling only fn T = 1, and sl v r t is the one where 
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• D0 5.4fb" 1 
— R v tb+tqb 

■ -■L T tb+tqb 
R T tb+tqb 

L v tb+tqb 

■ W+jets 

Multijets 
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P T (I) [GeV] 




top frame 



1 -0.5 
Cos(b-tagged j 



top frame 



FIG. 2: Comparison of the SM backgrounds and data for selected discriminating variables with all channels combined: (a) 
lepton pr, (b) cosine of the angle between the lepton (in the reconstructed top quark frame) and the reconstructed top quark (in 
the center of mass frame) and (c) cosine of the angle between the leading fa-tagged jet and the lepton (both in the reconstructed 
top quark frame) . Superimposed are the distributions from single top quark production ( "ifa + tqb" ) with one non- vanishing 
non-SM coupling (all other couplings set to zero) normalized to 10 times the SM single top quark cross section. The VK+jets 
contributions include the smaller backgrounds from Z+jets and dibosons. 
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FIG. 3: The left column shows BNN discriminant output dis- 
tributions of data and of the sum of the SM backgrounds 
with all channels combined for the whole discriminant range; 
superimposed are the distributions for the single top quark 
contributions scaled by (a) 5 times for the {Lv, Lt) scenario, 
(c) 20 times for the {Lv, Rv) scenario, and (e) 5 times for the 
{Lv, Rt) scenario. The right column shows BNN discrimi- 
nant output distributions in the high discriminant region for 
(b) the {Lv, Lt) scenario, (d) the {Lv, Rv) scenario, and 
(f) the {Lv, Rt) scenario. The hatched bands give the un- 
certainty on the background sum. The W+jets contributions 
include the smaller backgrounds from Z+jets and dibosons. 



both couplings Jl v = 1 and fu T = 1. The last sam- 
ple is indicated as "Ly + Rt" in Fig. |3f. We assume 
a Poisson distribution for data counts and uniform prior 



TABLE II: One-dimensional upper limits at 95% C.L. for 
anomalous Wtb couplings in the three scenarios. 



Scenario 



Cross section 



Coupling 



{Lv,Lt) 
{Lv,Rv) 
{Lv,Rt) 



< 0.60 pb 

< 2.81 pb 

< 1.21 pb 



\V tb -Sl t f < 0-06 
V i6 • /« v . | 2 < 0.93 
\V tb -f RT \ 2 <0.13 



probability for nonncgative values of the SM and non-SM 
couplings. The output discriminants for the signal, back- 
grounds, and data are used to form a binned likelihood 
as a product over all six analysis channels and all bins, 
taking into account all systematic uncertainties and their 
correlations. The expected posterior probabilities are ob- 
tained by setting the number of data counts to be equal 
to the predicted sum of the signal and backgrounds. 

Figure U shows the 2D posterior probability density 
distributions for the three scenarios. We do not ob- 
serve significant deviations from the SM expectations 
and therefore compute 95% C.L. upper limits on the 
anomalous couplings by integrating out the left-handed 
vector coupling to get a one-dimensional posterior prob- 
ability density. The measured values are given in Ta- 
ble |TTJ With the SM constraint on the left-handed vector 
coupling, i.e. \Vtb • Jl v \ = 1, the 95% C.L. limits on 
left-handed tensor, right-handed vector and tensor cou- 
plings are \V tb ■ J Lt \ 2 < 0.05, \V tb ■ f Rv \ 2 < 0.50 and 
\V t b ■ fn T \ 2 < 0.11, respectively. 

In summary, we have presented a search for anomalous 
Wtb couplings using 5.4 fb -1 of DO data in the single 
top quark hnal state. We find no evidence for anomalous 
couplings and set 95% C.L. limits on these couplings. 
These represent improvements in the limits by factors of 
2.6 to 5.0 in terms of couplings squared compared to the 
previous results [H[ while a factor of approximately 2.5 is 
expected from the increase in integrated luminosity. This 
result represents the most stringent direct constraints on 
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FIG. 4: Two-dimensional posterior probability density distributions for the anomalous couplings. The left row (a) shows the 
distribution for the (Lv, Lt) scenario, the middle row (b) for the (Lv, Rv) scenario, and the right row (c) for the (Lv, Rt) 
scenario. The dots represent the peak posterior from our data in comparison with the SM predictions. 



anomalous Wtb interactions. 

We thank the staffs at Fermilab and collaborating 
institutions, and acknowledge support from the DOE 
and NSF (USA); CEA and CNRS/IN2P3 (France); 
FASI, Rosatom and RFBR (Russia); CNPq, FAPERJ, 
FAPESP and FUNDUNESP (Brazil); DAE and DST (In- 
dia); Colciencias (Colombia); CONACyT (Mexico); KRF 
and KOSEF (Korea); CONICET and UBACyT (Ar- 
gentina); FOM (The Netherlands); STFC and the Royal 
Society (United Kingdom); MSMT and GACR (Czech 
Republic); CRC Program and NSERC (Canada); BMBF 
and DFG (Germany); SFI (Ireland); The Swedish Re- 
search Council (Sweden); and CAS and CNSF (China). 



[1] F. Abe et al. (CDF Collaboration), Phys. Rev. Lett. 74, 
2626 (1995). 

[2] S. Abachi et al. (DO Collaboration), Phys. Rev. Lett. 74, 
2632 (1995). 

[3] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 

103, 092001 (2009). 
[4] T. Aaltonen et al. (CDF Collaboration), Phys. Rev. Lett. 

103, 092002 (2009). 
[5] S. Cortese and R. Petronzio, Phys. Lett. B 253, 494 

(1991). 

[6] S. S. D. Willenbrock and D. A. Dicus, Phys. Rev. D 34, 
155 (1986). 

[7] C.-P. Yuan, Phys. Rev. D 41, 42 (1990). 
[8] N. Kidonakis, Phys. Rev. D 74, 114012 (2006). The 
cross sections for the single top quark processes (m t = 
172.5 GeV) are 1.04 ± 0.04 pb (s-channel) and 2.26 ± 

0.12 pb (t-channel). 

[9] V. M. Abazov et al. (DO Collaboration) . larXiv: 1 108.3091 1 
[hep-ph], submitted to Phys. Rev. D. 

[10] V. M. Abazov et al. (DO Collaboration). larXiv:1105. 27881 
[hep-ph], submitted to Phys. Lett. B. 

[11] G. L. Kane, G. A. Ladinsky, and C.-P. Yuan, Phys. Rev. 
D 45, 124 (1992); J. A. Aguilar-Saavedra, Nucl. Phys. B 
812, 181 (2009). 

[12] F. Larios, M. A. Perez, and C.-P. Yuan, Phys. Lett. B 
457, 334 (1999); G. Burdman, M. C. Gonzalez-Garcia, 
and S. F. Novaes, Phys. Rev. D 61, 114016 (2000); 



B. Grzadkowski and M. Misiak, Phys. Rev. D 78, 077501 
(2008); J. P. Lee and K. Y. Lee, Phys. Rev. D 78, 056004 
(2008), and references therein. 
[13] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 

100, 062004 (2008). V. M. Abazov et al. (DO Collabora- 
tion), Phys. Rev. D 83, 032009 (2011). 

[14] A. Abulencia et al. (CDF Collaboration) Phys. Rev. D 
73, 11103 (2006). A. Abulencia et al. (CDF Collabora- 
tion) Phys. Rev. D 75, 052001 (2007). T. Aaltonen et 
al. (CDF Collaboration) Phys. Lett. B 674, 160 (2009). 
T. Aaltonen et al. (CDF Collaboration) Phys. Rev. Lett. 
105, 042002 (2010). 

[15] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 
102, 092002 (2009). 

[16] E. Boos, L. Dudko, and T. Ohl, Eur. Phys. J. C 11, 473 
(1999). 

[17] D. O. Carlson, E. Malkawi, and C.-P. Yuan, Phys. Lett. 
B 337, 145 (1994); E. Malkawi and C.-P. Yuan, Phys. 
Rev. D 50, 4462 (1994); A. P. Heinson, A. S. Belyaev, 
and E. Boos, Phys. Rev. D 56, 3114 (1997). 

[18] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 

101, 221801 (2008). 

[19] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 
99, 191802 (2007); V. M. Abazov et al. (DO Collabora- 
tion), Phys. Lett. B 693, 81 (2010). 

[20] T. Aaltonen et al. (CDF Collaboration) Phys. Rev. Lett. 

102, 151801 (2009). 

[21] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 

102, 191802 (2009). 

[22] V. M. Abazov et al. (DO Collaboration), Phys. Lett. B 
641, 423 (2006); V. M. Abazov et al. (DO Collaboration), 
Phys. Rev. Lett. 100, 211803 (2008); V. M. Abazov et 
al. (DO Collaboration), Phys. Lett. B 699, 145 (2011). 

[23] T. Aaltonen et al. (CDF Collaboration) Phys. Rev. Lett. 

103, 041801 (2009). 

[24] V. M. Abazov et al. (DO Collaboration), Nucl. lustrum. 

Methods in Phys. Res. A 620, 490 (2010). 
[25] E. E. Boos et al, Phys. Atom. Nucl. 69, 1317 (2006). We 

use SINGLETOP version 4.2pl. 
[26] J. Pumplin et al., J. High Energy Phys. 07, 012 (2002). 

We use versions CTEQ6M for signal and CTEQ6L1 for 

background. 

[27] Z. Sullivan, Phys. Rev. D 70, 114012 (2004). 

[28] J. M. Campbell, R. Frederix, F. Maltoni, and F. Tramon- 



8 



tano, Phys. Rev. Lett. 102, 182003 (2009). 

[29] M. L. Mangano et al., J. High Energy Phys. 07, 001 
(2003). We use ALPGEN version 2.11. 

[30] T. Sjostrand, S. Mrenna, and P. Skands, J. High En- 
ergy Phys. 08, 026 (2006). We use version 6.409. 

[31] R. Brun and F. Carminati, CERN Program Library Long 
Writeup, Report No. W5013, 1993. 

[32] S. Moch and P. Uwer, Phys. Rev. D 78, 034003 (2008). 
At m t = 172.5 GeV, a(pp -+ti+X) = 7.46 pb. 



[33] R. K. Ellis, Nucl. Phys. Proc. Suppl. 160, 170 (2006). 

We use mcfm version 5.1. 
[34] R. M. Neal, Bayesian Learning for Neural Networks 

(Springer- Verlag, New York, 1996). 
[35] V. M. Abazov et al. (DO Collaboration), Phys. Rev. Lett. 

98, 181802 (2007). 
[36] V. M. Abazov et al. (DO Collaboration), Phys. Rev. D 

78, 012005 (2008). 



